[ILM] Add "wait for snapshot" policy field to Delete phase #68505

yuliacech · 2020-06-08T12:41:01Z

Summary

This PR adds a text field for "wait for snapshot policy" in Delete phase of an index lifecycle policy.

How to test

Create a repository for snapshots
Create a snapshot policy that backs up an index
Create an index lifecycle policy that deletes the managed index with "wait for snapshot" option
Create an index template with an associated lifecycle policy and a write index for rollover
Add 1 document to the index for rollover to happen (for testing we are setting max_docs: 1).
After conditions for Delete phase are met (for testing Delete phase immediately follows rollover) and snapshot policy is executed (for testing run the snapshot policy manually), the managed index is deleted.

Expand for console commands

(when running yarn es you need to add -E path.repo=./my_repo_test to be able to create a repository)

PUT /_snapshot/my_repo
{
  "type": "fs",
  "settings": {
    "location": "./my_repo_test",
    "compress": true
  }
}

(index to be backed up test-*, snapshots created once a year and deleted after 10 min)

PUT _slm/policy/my_snapshot_policy
{
  "name": "<snapshot-{now}>",
  "schedule": "0 0 0 1 1 ?",
  "repository": "my_repo",
  "config": {
    "indices": "test-*"
  },
  "retention": {
    "expire_after": "10m",
    "min_count": 1,
    "max_count": 3
  }
}

(rollover after 1 doc, delete after my_snapshot_policy created a snapshot).

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d",
            "max_docs": 1
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "0d",
        "actions": {
          "wait_for_snapshot": {
            "policy": "my_snapshot_policy"
          },
          "delete": {}
        }
      }
    }
  }
}

(update lifecycle poll interval)

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.core.indexlifecycle": "TRACE",
    "logger.org.elasticsearch.xpack.indexlifecycle": "TRACE",
    "logger.org.elasticsearch.xpack.core.ilm": "TRACE",
    "logger.org.elasticsearch.xpack.ilm": "TRACE",
    "indices.lifecycle.poll_interval": "10s"
  }
}

PUT _template/test
{
  "index_patterns": [
    "test-*"
  ],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0,
    "index.lifecycle.name": "my_policy",
    "index.lifecycle.rollover_alias": "test-alias"
  }
}
PUT /test-000001
{
  "aliases": {
    "test-alias": {
      "is_write_index": true
    }
  }
}

PUT test-alias/_doc/1
{
  "a": "a"
}

Release Note

We added a snapshot policy name field to Delete phase of index lifecycle policy. This option ensures that the snapshot policy is executed before the managed index is deleted.

…phase)

elasticmachine · 2020-06-08T15:14:23Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

yuliacech · 2020-06-08T17:41:30Z

As suggested by Alison, I'll be adding some tests to verify payload data.

cjcenizal

Tested locally, code LGTM! Great work Yulia. I'll review again after you've added tests.

I tested this by creating an ILM policy via the UI that resulted in this configuration:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "100s",
            "max_size": "50gb",
            "max_docs": 1
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "0d",
        "actions": {
          "wait_for_snapshot": {
            "policy": "test"
          },
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Then I created a policy that would snapshot everything and set it to execute in a couple of minutes (a little tricky because it shows timing according to GMT instead of my local timezone).

Then I followed the other quick-test steps from the ILM README to associate my policy with an index.

In a couple minutes, the snapshot was taken and then index was deleted as expected.

yuliacech · 2020-06-10T16:43:09Z

@elasticmachine merge upstream

yuliacech · 2020-06-10T16:56:02Z

Hey team,
I added some jest integration tests: for the scope of this PR, tests are only for Delete phase. But the setup for jest client integration is now in place to add more tests later.
Please let me know what you think!

cjcenizal

Tests LGTM! Thanks for adding those!

cjcenizal · 2020-06-10T19:18:45Z

The failing tests look unrelated to these changes. I don't see master failing CI so I'd try merging in the last version of master to see if a fix got merged.

yuliacech · 2020-06-11T08:54:24Z

@elasticmachine merge upstream

kibanamachine · 2020-06-11T10:38:50Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fc255ec

History

💔 Build #53092 failed b0bcf85
💚 Build #52371 succeeded dbdf8e9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (38 commits) Support migrating from reserved feature privileges (elastic#68504) add `preference` field to SavedObjectsFindOptions (elastic#68620) [ILM] Add "wait for snapshot" policy field to Delete phase (elastic#68505) Cleanup old license overwrites (elastic#68744) Bump TypeScript to v3.9 (elastic#67666) [APM] Service maps - adds new storybook stories to test out various data sets (elastic#68727) Fix vega specification parsing (elastic#67963) docs: add more api information (elastic#68717) [APM] Don't show annotations on charts with no data (elastic#68829) [Metrics UI] Fix Inventory View sorting by handling null values (elastic#67889) skip flaky suite (elastic#68836) [SIEM][Detections Engine] - Fix reference rule url overflow (elastic#68640) Index pattern public api => common (elastic#68289) [APM] Lazy-load alert triggers (elastic#68806) [DOCS] Fix table formatting in ingest manager settings (elastic#68824) [Endpoint] Functional Tests cleanup (elastic#68756) revert previous commit which was unintentional Use Github token instead for project assignments [SIEM][Exceptions] - ExceptionsViewer cleanup (elastic#68739) move @kbn/storybook to devDeps (elastic#68791) ...

…8505) * [ILM] Add "wait for snapshot" text field to edit policy form (Delete phase) * [ILM] Add jest client integration tests for delete phase Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…68892) * [ILM] Add "wait for snapshot" text field to edit policy form (Delete phase) * [ILM] Add jest client integration tests for delete phase Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[ILM] Add "wait for snapshot" text field to edit policy form (Delete …

dbdf8e9

…phase)

yuliacech requested a review from a team as a code owner June 8, 2020 12:41

yuliacech added Feature:ILM release_note:enhancement Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more labels Jun 8, 2020

yuliacech added v7.9.0 v8.0.0 labels Jun 8, 2020

yuliacech requested a review from cjcenizal June 8, 2020 17:15

cjcenizal approved these changes Jun 9, 2020

View reviewed changes

[ILM] Add jest client integration tests for delete phase

d096e57

Merge branch 'master' into wait_for_snapshot

b0bcf85

cjcenizal approved these changes Jun 10, 2020

View reviewed changes

Merge branch 'master' into wait_for_snapshot

fc255ec

yuliacech merged commit 4b1b475 into elastic:master Jun 11, 2020

yuliacech mentioned this pull request Jun 11, 2020

[7.x] [ILM] Add "wait for snapshot" policy field to Delete phase (#68505) #68892

Merged

cjcenizal mentioned this pull request Jun 24, 2020

Update the ILM UI to include the ability to wait for snapshot #56776

Closed

yuliacech deleted the wait_for_snapshot branch July 20, 2020 13:30

nzamps mentioned this pull request Aug 10, 2020

wait for rollup in ILM #74719

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ILM] Add "wait for snapshot" policy field to Delete phase #68505

[ILM] Add "wait for snapshot" policy field to Delete phase #68505

yuliacech commented Jun 8, 2020 •

edited

Loading

elasticmachine commented Jun 8, 2020

yuliacech commented Jun 8, 2020

cjcenizal left a comment

yuliacech commented Jun 10, 2020

yuliacech commented Jun 10, 2020

cjcenizal left a comment

cjcenizal commented Jun 10, 2020

yuliacech commented Jun 11, 2020

kibanamachine commented Jun 11, 2020

[ILM] Add "wait for snapshot" policy field to Delete phase #68505

[ILM] Add "wait for snapshot" policy field to Delete phase #68505

Conversation

yuliacech commented Jun 8, 2020 • edited Loading

Summary

How to test

Release Note

elasticmachine commented Jun 8, 2020

yuliacech commented Jun 8, 2020

cjcenizal left a comment

Choose a reason for hiding this comment

yuliacech commented Jun 10, 2020

yuliacech commented Jun 10, 2020

cjcenizal left a comment

Choose a reason for hiding this comment

cjcenizal commented Jun 10, 2020

yuliacech commented Jun 11, 2020

kibanamachine commented Jun 11, 2020

💚 Build Succeeded

History

yuliacech commented Jun 8, 2020 •

edited

Loading